Demo: Multiple Regression and Model Selection (Self-Paced)

In this demonstration, you use a SAS program to create a multiple regression model for Donation_Amt using the stepwise model selection method. You use Akaike's information criterion (AIC) to add and remove variables from the model. The program Multiple Regression.sas is added to the previously created flow Correlation and Regression.flw.

Reminder: If you restarted your SAS session, or it has timed out due to inactivity, you must re-create the course libraries, LOCALLIB and VST. To do this, run the AssignLibrary flow.

  1. In SAS Viya, launch SAS Studio by clicking the Applications menu button (the three-by-three grid in the upper left corner), expand ANALYTICS LIFE CYCLE, and click Develop Code and Flows.

  2. In the navigation panel on the far left, click the Explorer button (second from top). Expand Files >Home > workshop > VST. Double-click AssignLibrary.flw to open it.

  3. Click the Run button on the flow toolbar.

Demo Steps

  1. Open the previously created flow Correlation and Regression.flw from the VST folder. We will add a program to this flow.

  2. Navigate to the VST folder and double click on Multiple Regression.sas. Click on the SAS Program step and view the code. While the majority of the code is beyond the scope of this course, note the highlighted portions. The first highlighted section (param=glm) indicates the GLM coding of categorical variables. This default coding creates one numeric (dummy) variable for each level of a categorical predictor. Effect coding (param=effect) and reference coding (param=ref) are other available options. They create one fewer dummy variable than GLM coding and are used for comparing levels to the mean or to a reference level respectively.
    The second highlighted section (selection=stepwise (select=aic)) shows that the multiple regression model uses stepwise variable selection with variables being selected based on Akaike’s information criterion. Here is the code: 
    ods graphics / imagemap=on; 


proc glmselect data=VST.PVA_DONORS outdesign(addinputvars)=Work.reg_design 

plots=(criterionpanel); 

class DemHomeOwner DemGender DemCluster StatusCatStarAll StatusCat96NK/ param=glm; 

model Donation_Amt=GiftCnt36 GiftCntAll GiftCntCard36 GiftCntCardAll GiftAvgLast GiftAvg36 GiftAvgAll GiftAvgCard36 GiftTimeLast GiftTimeFirst PromCnt12 PromCnt36 PromCntAll PromCntCard12 PromCntCard36 PromCntCardAll DemAge DemMedHomeValue DemPctVeterans DemMedIncome DemHomeOwner DemGender DemCluster StatusCatStarAll StatusCat96NK/ showpvalues selection=stepwise (select=aic); 

run; 

 
proc reg data=Work.reg_design alpha=0.05 plots(only)=(diagnostics residuals dffits 

observedbypredicted); 

ods select ParameterEstimates DiagnosticsPanel ResidualPlot DFFITSPlot 

      ObservedByPredicted; 

model Donation_Amt=&_GLSMOD / vif; 

run;

  3. Click the + Code to Flow button and add the program to the previously saved Correlation and regression flow. Navigate to the flow tab of Correlation and regression.flw.

  4. Right-click on the Multiple regression node and choose Run node. Save the Correlation and regression flow.

  5. Click the Submitted Code and Results tab in the upper half of the work area to view the results.

An image of the results.
An image of the Class Level Information.

The multiple regression model used 3101 observations due to complete case analysis. The stepwise-AIC model has 26 potential effects and 88 potential parameters. The number of parameters is higher than the number of effects because of the dummy-coding necessary to account for the levels of the categorical predictors.

An image of the Stepwise Selection Summary.

The stepwise process started with a model containing only the intercept. GiftAvg36 was added at step 1 because it reduced AIC (that is, it improved model fit) more than any other single variable. For steps 2 through 6, adding variables improved AIC more than removing variables. At step 7, a previously entered variable, PromCntCard12, was removed. Then in steps 8–13, variables were added based on producing the greatest reduction in the model AIC.

An image of the Stop Details.

Model selection stopped at the optimal value of AIC. Any addition or reduction of a single variable from the model at step 13 increased the AIC and made a worse model by this criterion. The Stop Details table shows potential candidate variables to add or remove, both of which increase the AIC, but by a small amount (approximately 0.4).

An image of the Fit Criteria for Donation_Amt.

Fit criteria plots show on the X axis the 13 models corresponding to each step, and on the Y axis one of four fit statistics. Keep in mind that the models assessed during the stepwise selection process are not all possible models, but only a small subset. Of that subset, only fit statistics for the model present at each step are graphed. These graphs show that the final model had the best values of AIC, AICC, and adjusted R-square of any of the 13 models. The best values are indicated by a star. For the SBC, the best model was the model at step 10, which had three fewer inputs than the final AIC model.

An image of the Selected Model.
An image of the Analysis of Variance.

The selected model has 11 predictors and is statistically significant (P < 0.0001), meaning at least one predictor in the model has a slope that is significantly different from zero (assuming alpha = 0.05). The R-square indicates that this model explains 54% of the variability in Donation_Amt. The MSE is 302.96 (compare with MSE=392 in the simple linear regression) and the adjusted R-square equals 0.54 (compare with R-square = 0.42 in the simple linear regression). The multiple regression fits the data better than the simple linear regression model. A note indicates that the p-values have not been adjusted for the model selection process. This refers to the p-values being biased too low when used for automated model selection. (For more information about this, see Flom and Cassell 2007).

An image of the Parameter Estimates.

The Parameter Estimates table shows the estimates for the intercept and the slopes for each input variable. The two categorical variables, StatusCat96NK and StatusCatStarAll, each have multiple parameter estimates, one for each level of the variable except for the reference level. The reference level has a parameter estimate of zero. The other levels have parameters that describe the difference between that specific level and the reference level. For example, the parameter estimate of 2.4977 for StatusCat96NK A indicates a $2.4977 greater average donation amount for members of the A-group than for members of the S- group.

Notice that several of the effects in the model, such as PromCntCardAll and PromCntCard12 are not significant at the alpha=0.05 level. This is because the model selection approach was based on Akaike's information criterion and not significance level. At each step in the model-building process, a variable was either added or removed that improved the AIC (smaller is better) relative to the previous model. At the final step, no further additions or removals of one variable made the AIC smaller. Using other model selection criteria such as SBC, adjusted R-square, or significance level can result in different final models than the stepwise AIC model (not shown). Some of these other models are discussed later in this lesson. The diagnostics plots for this model are discussed in the next demonstration.